Cosmos: Adds Cross-Region Hedging Design Spec to Driver Crate by kundadebdatta · Pull Request #4330 · Azure/azure-sdk-for-rust

kundadebdatta · 2026-05-03T21:45:34Z

Summary

Adds HEDGING_SPEC.md to the azure_data_cosmos_driver crate's docs/ folder. This is a doc-only PR — no production code, no API changes, no test changes, no Cargo.toml changes. Single file, +1569 lines.

The spec is the design document for the cross-region hedging (AvailabilityStrategy / HedgingStrategy) feature that will be implemented in a follow-up series of PRs. It is being landed on the previews branch ahead of implementation so reviewers can iterate on the design in-tree alongside the companion specs already there.

Why hedging?

When a Cosmos DB region is degraded but not fully down (elevated tail latency, slow GC, partial network blip) the existing failover paths — PPAF (per-partition automatic failover) and PPCB (per-partition circuit breaker) — do not trigger because the region eventually returns successful responses. Applications see p99 / p99.9 latency spikes on requests routed to the slow region.

Cross-region hedging issues a speculative second request to an alternate region after a configurable threshold and returns whichever response arrives first, bounding tail latency at roughly threshold + cross-region-RTT.

Scope of this PR

In scope: Design document only.

Out of scope (follow-up PRs):

HedgingStrategy / AvailabilityStrategy types
should_hedge() / is_final_result() pure functions
execute_with_hedging() orchestrator
HedgeDiagnostics
Integration into cosmos_driver.rs
PPAF default-strategy auto-enable wiring
Unit + fault-injection tests

What the spec covers (17 sections, ~1,569 lines)

§	Topic
1	Goals, non-goals, and the phased operation-type rollout (Phase 1 reads + writes; Phase 2 Query + ReadMany; Phase 3 ChangeFeed + metadata; Future sprocs/triggers/UDFs)
2	Background: full walkthrough of the .NET v3 reference implementation (`CrossRegionHedgingAvailabilityStrategy`), including PPAF/PPCB integration
3	Architectural overview
4	Configuration surface (`HedgingStrategy`, builder, environment variables)
5	Eligibility rules (`should_hedge()` decision matrix) and default hedging enablement driven by PPAF with a full activation truth table
6	Hedging algorithm — primary at t=0, hedge fan-out at `threshold + N · step`, `tokio::select!` race, drain loop
7	Final-vs-transient status code classification (incl. explicit `403` / `403/3` rows)
8	Operation-pipeline integration, including the explicit local-only-retry contract (`ExcludeRegions` invariant for retries inside a hedge)
9	Interaction with PPAF, PPCB, session consistency, throughput control, end-to-end timeout
10	Diagnostics & observability (`HedgeDiagnostics` shape, attachment contract, *reserved `cosmos.hedge.` tracing/metrics surface**)
11	Options API design and layered resolution priority table (operation > client > SDK default > none)
12	Cancellation & resource cleanup (lock-free `tokio_util::sync::CancellationToken` hierarchy)
13	Multi-write region write hedging (409/412 risk, idempotency considerations)
14	Error handling & edge cases (RU accounting, late-hedge budget)
15	Test plan
16	Implementation phases
17	Open questions (most resolved during spec review)

Design highlights

PPAF-only auto-enable, matching .NET v3 exactly. Enabling PPCB alone
does not auto-enable hedging — PPCB is failure-driven and does not by
itself signal a desire for latency hedging.
Default thresholds when auto-enabled by PPAF:
min(1000ms, request_timeout / 2) / 500ms, mirroring .NET
(Java's static 500ms / 100ms is documented in the cross-SDK comparison
for reference but not adopted).
Lock-free design throughout — tokio_util::sync::CancellationToken
for cancellation, Arc<Bytes> for zero-copy hedge body sharing.
Type-safe disabled sentinel via AvailabilityStrategy::Disabled enum
variant rather than .NET's nullable TimeSpan? / sentinel object pattern.
Always-full HedgeDiagnostics attached whenever a strategy was active,
avoiding .NET's two-shape (fast-path vs drain-path) bookkeeping.
Explicit contracts that .NET and Java rely on implicitly: local-only
retry contract (§8.4), diagnostics attachment contract (§10.1),
preferred-regions precondition (§5.2).
Reserved telemetry namespace under cosmos.hedge.* for tracing events
and metrics, ready for a future observability PR without breaking changes.

Cross-SDK alignment

The spec was reviewed against both the .NET v3 (CrossRegionHedgingAvailabilityStrategy) and Java v4 (ThresholdBasedAvailabilityStrategy) implementations. Final alignment summary:

Capability	.NET v3	Java v4	Rust spec
Auto-enabled by PPAF	✅	✅	✅
Auto-enabled by PPCB alone	❌	❌	❌
Default thresholds	`min(1000ms, RT/2)` / `500ms`	`500ms / 100ms`	`min(1000ms, RT/2)` / `500ms` (= .NET)
Phase-1 read hedging	✅	✅	✅
Phase-1 write hedging on multi-master	✅	✅	✅
Query / ReadMany	✅	✅	Phase 2
ChangeFeed + metadata	✅ / ❌	❌ / ❌	Phase 3

Validation

Markdown only — renders correctly in GitHub preview
All cross-SDK source citations link to specific file/line ranges on
Azure/azure-cosmos-dotnet-v3 (main) and Azure/azure-sdk-for-java (main)
All internal §N.N references resolve to existing spec sections
No .rs, .toml, Cargo.lock, or test changes; CI affecting only the
doc-render job

Branch / target

Source branch: users/kundadebdatta/3935_add_hedging_spec
Target branch: release/azure_data_cosmos-previews
Commits: 2 (Code changes to add hedging spec. →
Code changes to update hedging spec.)
Files changed: 1
Diff: +1,569 / -0

Follow-up work tracker

The implementation will land in subsequent PRs roughly tracking §16 phases:

Phase 1 — HedgingStrategy types + should_hedge() + is_final_result()
- orchestrator + reads/writes (multi-master) + PPAF auto-enable
Phase 2 — Query / ReadMany hedging
Phase 3 — Change feed / metadata operation hedging
Future — sprocs/triggers/UDFs, adaptive thresholds, telemetry surface

NaluTripician

Cross-checked the spec against the actual .NET source in azure-cosmos-dotnet-v3 (CrossRegionHedgingAvailabilityStrategy.cs, AvailabilityStrategy.cs, DocumentClient.InitializePartitionLevelFailoverWithDefaultHedging). Overall the spec is well-grounded — author clearly read the .NET code, not just the docs. One material issue around the SDK-default write-hedging behavior on multi-master, plus a handful of minor nits inline.

…datta/3935_add_hedging_spec

Copilot

Pull request overview

Adds a new in-repo design specification (HEDGING_SPEC.md) to the Cosmos driver crate docs, describing the planned cross-region hedging (availability strategy) feature and its intended integration with existing routing/retry systems.

Changes:

Adds a comprehensive design spec for cross-region request hedging (configuration, eligibility, algorithm, diagnostics, and phased rollout plan).
Documents intended interactions with PPAF/PPCB, session consistency, throughput control, deadlines, and cancellation.

…datta/3935_add_hedging_spec

This reverts commit 19f83c9.

FabianMeiswinkel

LGTM except for few comments that could be addressed after merging the PR.

…mbing Implements Part 1 of the cross-region hedging design spec (HEDGING_SPEC.md, merged via PR #4330). Pure data types and options plumbing only — no behavior change to the operation pipeline. Changes - New `src/options/availability_strategy.rs` with the public types from spec sections 4.1 and 4.2: `HedgeThreshold` (non-zero newtype), `HedgingStrategy`, and `AvailabilityStrategy`. - `OperationOptions` gains a public `availability_strategy: Option<AvailabilityStrategy>` field (spec section 4.3). The `CosmosOptions` derive auto-generates the matching `with_availability_strategy` builder method and view accessor, so the field participates in the existing environment → runtime → account → operation layered resolution. - `src/options/env_parsing.rs` gains `parse_hedging_threshold_from_env` and `parse_hedging_disabled_from_env` helpers for the two env vars defined in spec section 4.4. Both are gated `#[allow(dead_code)]` — they are consumed by the resolver introduced in Part 3. - `src/options/mod.rs` re-exports the three new public types. - New `docs/HEDGING_IMPLEMENTATION_PLAN.md` captures the 7-part rollout plan that this commit is Part 1 of. Account-level configuration uses the existing layered model: callers set `availability_strategy` on an `OperationOptions` value and pass it to `DriverOptionsBuilder::with_operation_options`. No new field on `DriverOptions` is required. Tests - 17 new unit tests covering `HedgeThreshold` invariants, builder round-trip, layered view resolution, and env-var parsing edge cases (unset, zero, non-numeric, empty, whitespace, case-insensitive matching). `cargo test -p azure_data_cosmos_driver --lib options::` reports 105 passed / 0 failed. - `cargo fmt`, `cargo build`, `cargo clippy --all-features --all-targets`, and `cargo doc --no-deps --all-features` all clean. Refs HEDGING_SPEC.md sections 4.1, 4.2, 4.3, 4.4, 11.3.1

Reconcile sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md with the now-merged Rust hedging design spec (PR Azure#4330, on release/azure_data_cosmos-previews) and align it with the cross-SDK Hedging Detection API contract. Key updates: - Describe the cross-SDK contract generically (no SDK-specific naming) and add a Rust-to-contract mapping table. - requested_regions() now returns Vec<RequestedRegion> carrying a RequestedRegionReason, conforming to the contract's per-region-reason shape; add the public RequestedRegion struct and RequestedRegionReason enum plus a From<ExecutionContext> mapping. This resolves the previously-deferred per-region-reason question. - Correct the Retry to OperationRetry rename mechanics to the real code: ExecutionContext derives Serialize only (no Deserialize), and the hand-written as_str() must also be updated; serde(alias) is now conditional on adding Deserialize. - Fix the HedgeDiagnostics reconciliation: strategy_config and response_region are non-optional in the merged HEDGING_SPEC.md; fix the hedging_started() doc-comment that called as_ref() on a non-Option. - Correct the surface inventory: real dispatch-site paths, Serialize-only derive, and that regions_contacted() is sorted/deduped (so new accessors are needed for dispatch/completion order with duplicates). - Complete the file, which was truncated mid-section 8.2: finish the test plan and add section 9 (CHANGELOG drafts) and section 10 (open questions). Spec-only; no production code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Code changes to add hedging spec.

203a7b5

github-actions Bot added the Cosmos The azure_cosmos crate label May 3, 2026

github-project-automation Bot added this to CosmosDB Rust SDK and Driver May 3, 2026

github-project-automation Bot moved this to Todo in CosmosDB Rust SDK and Driver May 3, 2026

Code changes to update hedging spec.

ca6d67b

kundadebdatta changed the title ~~Code changes to add hedging spec.~~ Cosmos: Adds Cross-Region Hedging Design Spec to Driver Crate May 4, 2026

kundadebdatta self-assigned this May 4, 2026

kundadebdatta added the high-availability label May 4, 2026

kundadebdatta moved this from Todo to In Progress in CosmosDB Rust SDK and Driver May 4, 2026

NaluTripician reviewed May 5, 2026

View reviewed changes

kundadebdatta added 2 commits May 5, 2026 10:56

Merge branch 'release/azure_data_cosmos-previews' into users/kundadeb…

55b1331

…datta/3935_add_hedging_spec

Code changes to update the spec doc.

ba4d0a6

kundadebdatta marked this pull request as ready for review May 5, 2026 22:58

kundadebdatta requested a review from a team as a code owner May 5, 2026 22:58

Copilot AI review requested due to automatic review settings May 5, 2026 22:58

Copilot started reviewing on behalf of kundadebdatta May 5, 2026 22:59 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

kundadebdatta and others added 2 commits May 6, 2026 19:12

Code changes to update spec to match the review comments.

0c21f8b

Merge branch 'release/azure_data_cosmos-previews' into users/kundadeb…

3647b40

…datta/3935_add_hedging_spec